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Adaptive  Visual  Servo  Regulation  Control  for  Camera-in-Hand 
Configuration  with  a  Fixed-Camera  Extension 

Enver  Tatlicioglu*.  Darren  M.  Dawson,  and  Bin  Xian 


Abstract:  In  this  paper,  image-based  regulation  control  of 
a  robot  manipulator  with  an  uncalibrated  vision  system  is  dis¬ 
cussed.  To  compensate  for  the  unknown  camera  calibration 
parameters,  a  novel  prediction  error  formulation  is  presented. 
To  achieve  the  control  objectives,  a  Lyapunov-based  adaptive 
control  strategy  is  employed.  The  control  development  for 
the  camera-in-hand  problem  is  presented  in  detail  and  a 
fixed-camera  problem  is  included  as  an  extension. 

I.  Introduction 

The  use  of  computer  vision  data  to  control  the  motion 
of  a  robot  manipulator  is  commonly  referred  to  as  visual 
servo  control.  For  single  camera  systems,  the  vision  data 
may  be  acquired  from  the  camera  which  is  mounted  at  the 
end-effector  of  the  robot  manipulator  (camera-in-hand)  or 
the  camera  may  be  fixed  in  the  workspace  (fixed-camera) 
so  that  it  can  observe  the  motion  of  the  end-effector  of 
the  robot  manipulator.  For  both  camera-in-hand  and  fixed- 
camera  systems,  the  two  dominant  control  architectures  are 
image-based  visual  servo  control  and  position-based  visual 
servo  control  [1].  For  image-based  control  schemes,  the 
Jacobian  matrix,  which  maps  the  image  errors  onto  the 
joint  space  of  the  manipulator  is  commonly  referred  as 
the  interaction  matrix.  The  interaction  matrix  is  a  nonlinear 
function  of  the  intrinsic  and  extrinsic  camera  calibration 
parameters.  Hence,  the  performance  of  the  controller  depends 
on  the  accurate  knowledge  of  the  camera  parameters  [2]. 
However,  camera  calibration  is  tedious,  difficult  and  costly 
[2],  [3].  On  the  other  hand,  a  control  scheme  with  off-line 
identification  of  the  camera  calibration  parameters  is  usually 
not  robust  to  the  change  of  parameters,  disturbances,  and 
unknown  environments  [4]. 

Beginning  from  the  early  1990’s,  the  focus  of  much  of  the 
research  on  visual  servoing  has  moved  to  uncalibrated  vision 
systems.  Yoshimi  and  Allen  [5]  utilized  the  geometric  effect 
of  rotational  invariance  to  estimate  the  interaction  matrix. 
Hosoda  and  Asada  [4]  presented  an  extended  least  squares 
algorithm  with  exponential  data  weighting  for  estimating  the 
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interaction  matrix.  In  [6],  Fakhry  and  Wilson  presented  mod¬ 
ifications  of  the  resolved  acceleration  controller  for  visual 
servoing.  Jagersand  et  al.  [7]  proposed  an  adaptive  visual 
servoing  controller.  In  [7],  a  nonlinear  least-squares  opti¬ 
mization  method  using  a  trust  region  method  and  Broyden 
estimation  is  utilized.  The  method  proposed  in  [7]  is  similar 
to  the  ones  in  [6]  and  [4].  Bishop  and  Spong  [8]  presented 
a  sampled-data  controller  for  uncalibrated  monocular  visual 
servo  systems  with  an  online  calibration  extension.  In  [9], 
Ruf  et  al.  proposed  an  online  calibration  algorithm  for 
position-based  visual  servo  control.  In  [10]  and  [11],  Pa- 
panikolopoulos  et.  al.  proposed  an  algorithm  based  on  online 
estimation  of  the  relative  distance  of  the  target  with  respect  to 
the  camera.  In  [12],  Malis  proposed  a  visual  servo  controller 
which  is  robust  to  changes  in  the  intrinsic  camera  calibration 
parameters.  Piepmeier  et  al.  [13]  proposed  a  dynamic  quasi- 
Newton  method  for  visual  servo  control  of  uncalibrated 
robotic  systems.  In  [13],  where  a  recursive  least  squares 
algorithm  is  utilized  to  estimate  the  unknown  interaction 
matrix.  In  [14],  Lu  et  al.  presented  an  online  algorithm  using 
the  least  square  method  to  calculate  the  extrinsic  orientation 
matrix.  In  [15],  Hespanha  et  al.  developed  theoretical  anal¬ 
ysis  for  uncalibrated  stereo  systems.  In  [16],  a  visual  servo 
controller  is  presented  for  end-effector  regulation  tasks  in 
the  presence  of  uncertain  camera  calibration  parameters.  In 
[17],  Kelly  et  al.  suggested  two  controllers  based  on  the 
transpose  Jacobian  control  philosophy.  However,  the  first 
controller  requires  the  depth  information  for  all  the  feature 
points,  and  the  second  controller  depends  on  the  approximate 
Jacobian  method  which  utilizes  the  best  available  information 
on  the  depth  and  the  camera  calibration  parameters.  Recently, 
Liu  et  al.  [2]  presented  adaptive  controllers  for  uncalibrated 
fixed-camera  systems.  The  first  controller  tracks  only  one 
feature  point  and  the  second  controller  can  track  multiple 
feature  points.  However,  with  a  six  degree-of-freedom  robot 
manipulator,  the  proposed  controller  can  track  at  most  three 
feature  points.  This  constitutes  a  problem  based  on  the  well- 
known  fact  that  four  coplanar  feature  points  on  an  object  are 
needed  to  determine  its  posture  from  their  projection  in  the 
image  plane.  Similar  to  [2],  the  development  in  [18]  is  also 
for  tracking  control  of  one  feature  point. 

In  this  paper,  image-based  regulation  control  of  a  robot 
manipulator  with  an  uncalibrated  vision  system  (i.e.,  the 
intrinsic  and  extrinsic  camera  parameters  are  unknown)  is 
addressed.  To  compensate  for  the  unknown  camera  calibra¬ 
tion  parameters,  a  prediction  error  formulation  is  presented. 
For  all  the  feature  points,  the  interaction  matrix  and  the  depth 
are  both  linearly  parameterized,  which  is  then  followed  by 


a  novel  prediction  error  formulation  to  design  the  nonlinear 
estimation  law.  The  novelty  of  this  formulation  over  the  past 
research  is  its  unique  method  to  linearly  parameterize  the 
interaction  matrix  and  the  depth  simultaneously.  To  develop 
the  error  system,  the  image  error  signals  for  four  feature 
points  are  combined  to  form  the  final  error  signal  which 
is  followed  by  the  Lyapunov-based  stability  analysis.  The 
novelty  of  this  analysis  is  the  design  of  the  Lyapunov  func¬ 
tion  which  incorporates  the  depth  informations  for  all  feature 
points.  Satisfaction  of  persistent  excitation  (PE)  conditions 
allows  the  image  and  the  estimation  error  signals  to  be  driven 
to  zero.  The  control  development  for  the  camera-in-hand 
problem  is  presented  in  detail  and  a  fixed-camera  problem 
is  included  as  an  extension. 

II.  Adaptive  Control  for  Camera-in-Hand 
Configuration 


A.  Geometric  Model 

To  make  the  subsequent  development  more  tractable,  four 
coplanar  target  points  located  on  a  static  object,  denoted 
by  Oi  Vz  =  1,..,4  are  considered.  In  order  to  develop 
a  geometric  relationship  between  the  fixed  object  and  the 
moving  camera,  an  inertial  coordinate  frame,  denoted  by 
X,  attached  to  the  object,  an  orthogonal  coordinate  frame, 
denoted  by  T,  whose  origin  coincides  with  the  optical  center 
of  the  moving  camera,  an  inertial  coordinate  frame,  denoted 
by  W,  attached  to  the  base  frame  of  the  robot  manipulator, 
and  an  orthogonal  coordinate  frame,  denoted  by  E,  attached 
to  the  end-effector  of  the  robot  manipulator  are  defined  (see 
Figure  1).  Let  the  3D  coordinates  of  the  ith  feature  point  on 
the  object  be  denoted  as  the  constant  xPi  £  R3  relative  to 
the  base  frame  W,  and  fhi  ( t )  £  R3  relative  to  T,  which  is 
defined  as  follows 

fhi  =  [  Xi  yi  Zi  ]T .  (1) 


robot  manipulator)  (Base  of  the  robot  manipulator) 


In  the  subsequent  development,  it  is  assumed  that  the 
object  is  always  in  the  field  of  view  (fov)  of  the  camera, 
hence  the  distances  from  the  origin  of  X  to  all  feature 
points  remain  positive  (i.e.,  Z;L  (t)  >  e  where  e  £  M  is  an 
arbitrarily  small  positive  constant)  and  bounded.  To  relate 
the  coordinate  systems,  let  R(t)  £  SO  ( 3)  and  Xf(t)  £ 
R3  denote  the  rotation  matrix  and  the  translation  vector 
respectively,  between  T  and  X.  Let  mi  (t)  £  M3  denote  the 
normalized  Euclidian  coordinates  for  the  ith  feature  point, 
which  is  defined  as  follows 

TO*  =  —fhi.  (2) 

Zi 

In  the  image  captured  by  the  camera,  each  of  these  feature 
points  have  projected  pixel  coordinates  expressed  relative  to 
X,  denoted  by  pi  (t)  £  R2,  defined  as  follows 

Pi=[  Ui  Vi  ] T  (3) 

where  u,  (f),  Vi  (t)  £  R.  The  projected  pixel  coordinates  of 
the  feature  points  are  related  to  the  normalized  Euclidian 
coordinates  by  the  pin-hole  model  of  [19]  such  that 

Pi  =  Ami  (4) 


where  A  £  R2x3  is  the  unknown  truncated  camera  calibra¬ 
tion  matrix  [20]  which  is  assumed  to  be  of  the  following 
form 


A4 


fku  cot  <j> 
fK 

sin  (j> 


U0 

Vo 


(5) 


where  ku,  kv  £  R  denote  camera  scaling  factors,  uq,  vq  £  R 
represent  the  pixel  coordinates  of  the  principal  point,  <j>  £  R 
is  the  angle  between  the  camera  axes,  and  f  £  R  is  the 
camera  focal  length. 


B.  Open-Loop  Error  System 

To  facilitate  the  open-loop  error  system  development, 
image  error  for  the  ith  feature  point,  denoted  by  e*  (t)  £  R2, 
is  defined  as  follows 


e,  =  Pi-  Pdi  (6) 

where  p^  £  R2  is  the  constant  desired  image  coordinates 
for  the  ith  feature  point.  The  dynamics  of  the  image  error  is 
found  as  follows 


e»  =  Pi  (7) 

—  Aei  Ttli  (8) 

Zi 

where  Aei  (t)  £  R2x3  is  a  function  of  camera  intrinsic  cal¬ 
ibration  parameters  and  image  coordinates  of  the  i*  feature 
point  as  shown  below 


Aei  =  A- 


0  0  rti 

0  0  Ui 


(9) 


and  from  Figure  1,  mi  ( t )  can  be  found  to  be  of  the  following 
form 


S  [Rr  ( Xpi  -  xr)]  ]  Jrq  (10) 


Fig.  1.  Geometric  relationships  between  the  fixed  object,  robot  manipulator, 
and  the  camera  attached  to  its  end-effector. 


TYli —  Rc  ^  R'l 


where  the  forward  kinematics  of  the  robot  manipulator  was 
utilized  [21].  In  (10),  Rc  £  SO  (3)  is  the  camera  extrinsic 
calibration  matrix,  Rr  (t)  £  SO  (3)  is  the  orientation  matrix 
of  the  end-effector  of  the  robot  manipulator,  Jr  ( q )  gl6x" 
is  the  Jacobian  matrix  of  the  robot  manipulator,  xr  (t)  £  R3 
is  the  end-effector  position  of  the  robot  manipulator  relative 
to  W,  q  (t)  £  R6  represents  the  joint  positions  of  the  robot 
manipulator,  and  S  (•)  £  R3x3  is  the  skew-symmetric  matrix 
form  of  its  argument  defined  as  follows 


- 1 
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^  1 

"  " 

£3 
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The  joint  velocity  vector  q  (t)  £  R6  is  assumed  to  be  the 
kinematic  control  input  such  that 

u  =  q.  (12) 

After  utilizing  (10)  and  (12),  the  dynamics  for  the  image 
error  of  (8)  can  be  rewritten  as  follows 

Zi&i  =  AeiRc  [  Rr  S  [Rr  (%pi  ^V)]  ]  Jr  U ■  (13) 

Remark  1:  In  [22],  it  was  shown  that  four  non-collinear 
object  feature  points  are  sufficient  to  determine  the  end- 
effector  frame  pose  with  respect  to  the  base  frame.  Based  on 
this  fact,  the  analysis  in  this  paper  will  be  based  on  regulation 
of  four  feature  points. 

Remark  2:  In  the  subsequent  analysis,  it  is  assumed  that 
the  end-effector  position  of  the  robot  manipulator  xr  (t), 
the  orientation  matrix  of  the  end-effector  of  the  robot 
manipulator  Rr  (t),  and  the  Jacobian  matrix  of  the  robot 
manipulator  Jr  (q)  are  known,  and  the  camera  intrinsic  and 
extrinsic  calibration  matrices  (i.e.,  Rc  and  A)  are  constant 
and  unknown. 

Remark  3:  In  the  subsequent  analysis,  it  is  assumed  that 
the  joint  positions  of  the  robot  manipulator  are  bounded  (i.e., 
q  ( t )  £  Coo )  provided  that  the  projected  pixel  coordinates  of 
all  feature  points  are  bounded  (i.e.,  pt  (t)  £  Coo  V*  =  1, 4). 

Remark  4:  In  the  subsequent  analysis,  it  is  assumed  that 
the  orientation  matrix  of  the  end-effector  of  the  robot 
manipulator  Rr  (t)  and  the  Jacobian  matrix  of  the  robot 
manipulator  Jr  (q)  are  bounded  signals  provided  that  the 
joint  positions  of  the  robot  manipulator  are  bounded. 

After  utilizing  (1)  and  (10),  the  dynamics  of  Zi  ( t )  are 
obtained  as  follows 

Zi  =  V c 3  [  Rr  S  [Rr  [Xpi  tXrf\  ]  Jr'U  (14) 

where  £  Rlx3  is  the  third  row  vector  of  the  extrinsic 

camera  calibration  matrix.  To  facilitate  the  subsequent  anal¬ 
ysis  a  diagonal  matrix,  denoted  by  Z  ( t )  £  R8x8,  with  its 
entries  being  Zi  (t)  Mi  =  1, 4,  and  a  combined  error  signal 
eft)  £  R8  are  defined  as  follows 

Z  =  diag  {z4,  z4,  z2,  z2,  z3,  z3,  z4,  z4j  (15) 


We  can  see  from  (15)  and  (16)  that  the  product  Z  (t)  e  (t)  is 
equal  to  the  following  expression 

Ze=[  zief  z2e2  z3e3  z4e\  ]T.  (17) 

After  utilizing  (13),  (15)  and  (16),  the  following  expression 
can  be  obtained  for  Z  (t)  e  (t) 

Ze  =  [  Zle\  z2ef  z3e\  z4e4  ]T  (18) 

=  B\Jru.  (19) 

where  B \  ( t )  £  R8x6  is  defined  in  Appendix  I.  After  adding 
and  subtracting  the  term  ^Z  (t)  e  ( t )  to  the  right-hand-side 
of  (19),  the  following  expression  can  be  obtained 

Ze  =  “2^  W  e  (^)  +  Hu  (20) 

where  II  (f)  £  R8x6  is  defined  in  Appendix  I.  Since,  the 
auxiliary  matrix  II  ( t )  has  unknown  constant  parameters,  the 
product  II  (t )  u  ( t )  can  be  linearly  parameterized  as  follows 

Uu  =  W3Q  (21) 

where  W3  ( t )  £  R8xp  is  a  known  regressor  matrix,  and  0  £ 
Rp  is  an  unknown  constant  parameter  vector1 .  The  estimation 
form  of  (21)  can  be  defined  as  follows 

flu  =  W3&  (22) 

where  n  (f)  £  R8x6  is  the  estimate  of  II  (t),  and  0  (t)  £  Rp 
is  the  yet  to  be  defined  dynamic  estimate  of  0.  After  adding 
and  subtracting  II  ( t )  u  ( t )  to  the  right-hand-side  of  (20),  the 
following  open-loop  error  system  is  obtained 

Ze=  -^Z(t)e{t)  +  flu  +  W36  (23) 

where  the  following  expression  was  utilized 

f[u  =  W30  (24) 

with  fl(t)  £  R8x6  being  defined  as  follows 

n  =  n  -  ri  (25) 

and  0  (t)  £  Rp  is  the  estimation  error  defined  as  follows 

0  =  0-0.  (26) 

C.  Closed-Loop  Error  System 

Based  on  the  subsequent  stability  analysis,  the  control 
input  u  ( t )  is  designed  as  follows 

u  =  —kflT  e  (27) 

where  k  £  R  is  a  positive  constant  control  gain.  After 
substituting  (27)  into  the  open-loop  error  system  in  (23),  the 
following  closed-loop  error  system  is  obtained 

Ze  =  ~Z{t)e(t)-kfmTe  +  W3Q.  (28) 

lrThe  reader  is  referred  to  [23]  for  the  derivation  of  W3  (t)  and  ©. 


D.  Prediction  Error  Formulation 

In  this  section,  a  prediction  error  formulation  for  the 
unknown  parameters  will  be  introduced.  From  Figure  1, 
fhi  (t)  can  be  written  as  follows2 

ttlj  —  Rc  \Rr  ('ttpi  Xr)  T"  Pc\  (29) 

where  pc  £  R3  is  the  position  of  the  origin  of  frame  T  with 
respect  to  frame  W  expressed  in  frame  T .  After  utilizing  (1), 
(2),  and  (4),  the  pixel  coordinates  for  the  ith  feature  point  can 
be  written  as  follows 

Vi  =  -^ARc  [rT  ( Xpi  -  xr)  +Pc]  (30) 

Zi 

where  the  corresponding  depth  can  be  written  as  follows 


Zi  —  rc3  \Rr  ( Xpi  xr )  -f  pcj  . 


(31) 


where  r^3  £  R3  is  the  last  row  of  Rc.  It  should  be  noted 
that,  in  (30)  and  (31),  A,  Rc,  xPi ,  pc  are  unknown  constant 
parameters,  and  xr  ( t ),  Rr  ( t )  are  measurable  signals  (see 
Remark  2).  Based  on  these  facts,  pi  ( t )  can  be  linearly 
parameterized  as  follows 


iyi9i, 

w2e2i 


(32) 


where  the  following  linear  parameterization  of  Zi  (t)  was 
utilized 


Zi  =  W202l. 


(33) 


We  note  that  Z{  (t)  are  assumed  to  satisfy  the  following 
inequalities 


Pi  (•)  >  Zi  it)  =  W2  ( t )  02l  >  >  0  (34) 

where  pi  (to,)  £  R  Vi  are  positive  functions  and  Si  £  R  Vi 
are  positive  constants.  In  (32)  and  (33),  W\  (t)  £  R2xri, 
W2  (t)  £  Rlxr2  are  measurable  regression  matrices,  and 
0 1 ,  £  Rri,  02,  £  Rr2  are  unknown  constant  parameter 
vectors3.  After  multiplying  both  sides  of  (32)  with  the  term 
W2  (t)  02,  the  following  expression  can  be  obtained 

PiW2e2i  =  WiQu-  (35) 


The  estimation  forms  of  (32)  and  (35)  can  be  defined  as 
follows 


Wi0ii 
Pi  =  - s - 

W202j 

PiW2Q  2,  =  VFi0h 


(36) 

(37) 


where  0h  (t)  £  Rri  and  02,  (t)  £  Rr2  are  the  estimates 
for  0 1 ,  and  02i,  respectively4.  Subtracting  (37)  from  (35) 


2For  the  derivation  of  the  expression  for  m,  the  reader  is  referred  to 
Appendix  VII. 

"'The  reader  is  referred  to  [23]  for  the  derivation  of  W\  (t),  IV2  (t),  0ii, 
and  02i,  Vi  =  1, .., 4. 

4In  the  subsequent  analysis,  a  projection  algorithm  will  be  utilized  to 

make  sure  that  W2  (/  )  02i  (t)  is  always  greater  than  some  arbitrarily  small 

positive  constant. 


and  adding  and  subtracting  the  term  p,  (t)  W2  (t)  02,  to  the 
right-hand- side  results  in  the  following  expression 

=  mz* 1  &I']T  (38) 

where  Wi(pi(t),t)  =  [  W1  ~PiW2  }  £  R2x(ri+r2)  is 
a  measurable  signal,  0j,  (f)  £  Rrj  is  the  estimation  error 
defined  as  follows 

%  =  Qji  -  Qji  Vi  =  1, 2,  Vi  =  1, 4  (39) 

and  the  prediction  error  for  the  ith  feature  point  p,j  (f)  £  R2 * 
is  defined  as  follows 

Pi— Pi- Pi-  (40) 

The  combination  of  the  pixel  coordinates  for  all  the  feature 
points,  denoted  by  pit)  £  R8,  is  defined  as  follows 

P=[Pi  P2  Pz  pJ  ]T  •  (41) 

The  prediction  error  p  (t)  £  R8  is  defined  as  follows 

p  =  p~p  (42) 

where  p(t)  £  R8  is  the  estimation  of  p(t).  Based  on  (38) 
the  prediction  error  p  (t)  can  be  written  as  follows 

p  =  F-iWQ  (43) 

where  F\  (•)  £  R8x8  is  an  auxiliary  matrix  defined  in 
Appendix  I,  and  W  (•)  £  R8xp  is  a  measurable  signal, 
0  £  Rp  is  the  combination  of  the  unknown  constants, 
0  (f)  €  Rp  is  the  estimation  of  0,  and  0  (t)  £  Rp  is  the 
estimation  error. 

Remark  5:  It  should  be  noted  that,  when  obtaining  (43) 
from  (38),  there  were  common  unknown  constants  for  dif¬ 
ferent  feature  points.  As  a  result  of  this  fact,  an  unknown 
vector  0  with  the  exact  same  size  as  in  (26)  is  obtained. 
Based  on  the  subsequent  stability  analysis,  the  estimation 

law  0  (t)  is  designed  as  follows 

0=  Proj  { aVWTp  +  TW^e)  (44) 

where  Proj{-}  is  defined  in  Appendix  VI,  and  a{t)  £  R  is 
a  positive  scalar  function  defined  as  follows 

a  =  l  +  =p(-)  (45) 

£ 

where  p  (•)  E  M  is  a  positive  function  defined  as  follows 

p(-)  =  max{p2(-)}  (46) 

and  e  £  R  is  a  positive  constant  defined  as  follows 

£  =  min  {£i}  ■  (47) 

i 

In  (44),  r  {t)  £  Rpxp  is  a  least-squares  estimation  gain 
matrix  designed  as  follows 

jt  (r-1)  4  2WtW.  (48) 

Remark  6:  It  should  be  noted  that  if  T_1  (to)  is  selected  to 
be  positive  definite  and  symmetric  then  T  (to)  is  also  positive 


definite  and  symmetric.  Therefore  it  follows  that  both  T_1  (t) 
and  T  ( t )  are  positive  definite  and  symmetric.  From  (48),  the 
following  expression  can  be  obtained 

r  =  -2YWtWT.  (49) 

From  (49),  it  is  clear  that  T  ( t )  is  negative  semidefinite; 
therefore,  V  (t)  is  always  constant  or  decreasing,  and  hence, 
it  follows  that  T  (t)  is  bounded  (for  more  details,  the  reader 
is  referred  to  [24]  and  [25]). 

E.  Stability  Analysis 

Theorem  1:  The  control  law  defined  in  (27)  and  the  up¬ 
date  law  defined  in  (44)  ensure  that  ||e  (£)||,  0  (i)  — >  0  as 
t  — »  +oo  provided  that  the  following  persistent  excitation 
conditions  [26]  hold 

rt0+T 

hh  <  /  n(r)nT(r )dr  <  7 2I8  (50) 

Jto 

rto+T  _ 

73Jp  <  /  WT (t)W {t)<It  <  74 Jp  (51) 

Jto 

where  £  K.  Vi  =  1, ..,  4  are  positive  constants,  /§  £  R8x8 
and  Ip  £  K.pxp  are  identity  matrices. 

Proof:  See  Appendix  IF 

F.  Conclusion 

The  image-based  regulation  control  problem  of  a  robot 
manipulator  with  an  uncalibrated  vision  system  was  ad¬ 
dressed.  The  depth  information  which  is  in  the  denominator, 
and  the  rest  of  the  interaction  matrix  were  simultaneously 
linearly  parameterized  for  the  first  feature  point.  After  utiliz¬ 
ing  a  novel  prediction  error  formulation  the  estimation  law 
was  designed.  To  avoid  the  singularity  issue  which  might  be 
caused  by  the  depth  signal  appearing  in  the  denominator, 
a  parameter  projection  algorithm  was  utilized.  Lyapunov- 
based  analysis  techniques  were  utilized  to  achieve  the  control 
objectives.  The  Lyapunov  function  was  designed  to  embody 
the  depth  information  of  all  feature  points.  This  design 
of  the  Lyapunov  function  allowed  us  to  have  a  depth-free 
stability  analysis.  Upon  satisfaction  of  persistent  excitation 
(PE)  conditions,  it  was  proven  that  both  the  image  and  the 
estimation  error  signals  are  driven  to  zero.  As  an  extension, 
a  fixed-camera  configuration  was  presented. 
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Appendix  I 

Auxiliary  Definitions 


The  auxiliary  matrix  Bi  (t).  introduced  in  (19),  is  defined 
as  follows 


AeiRc 

-Rr 

S  [Rr  {Xpl  -  Xr)\ 

Ae2Rc 

-Rr 

S  [Rr  (xp2  -  Xr)\ 

Ae3Rc 

-Rr 

S  [Rr  (xp3  -Xr)] 

Ae4Rc 

-Rr 

S  [Rr  (xp4  ~  Xr)] 

After  utilizing  (16)  and  the  time  derivative  of  (15),  the 
product  Z  (t)  e  (t)  can  be  written  as  follows 

Ze  =  Ei^  [  z i  zi  z2  z2  z3  z3  z4  z4  ]T  (53) 

where  the  diagonal  matrix  E4  ( t )  £  R8x8  is  defined  as 
follows 


Ei  —  diag  {en,  e42,  e2\ ,  e22,  e34,  e32 ,  e4±,  e42}  (54) 


with  dj  ( t )  being  the  jth  entry  of  d  (f).  The  second  term  on 
the  right-hand-side  of  (53)  can  be  written  as  follows 


dr  1  T 

—  [  ZI  Z2  Z2  Z3  z3  Z4  z4  \ 


C\Jru 


(55) 


where  (14)  was  utilized,  and  the  auxiliary  matrix  C\  ( t )  £ 
R8x6  is  defined  as  follows 


rj  h 

—  Rr 

S  [Rr  (^pl  ^r)] 

T 

rc3 

—  Rr 

S  [Rr  (%pl  ^r)] 

T 

rc3 

—  Rr 

S  [Rr  {%p2  ^V)] 

T 

rc3 

—Rr 

S  [Rr  (%p2  ^r)] 

T 

rc3 

—  Rr 

S  [Rr  ( Xps  ~  Xr)} 

T 

rc3 

—  Rr 

S  [Rr  (^p3  ^r)] 

T 

rc3 

—  Rr 

S  [Rr  (^p4  ^r)} 

T 

L  rc3 

—  Rr 

S  [Rr  ( Xp4  —  Xr)} 

The  auxiliary  signal  II  (f),  introduced  in  (20),  is 


follows 


II  =  (  i?!  +  -EiCi  )  Jr 


defined  as 


(57) 


where  (19),  (20),  (53)-(55)  were  all  utilized.  The  auxiliary 
matrix  Fj  (f)  e  R8x8,  introduced  in  (43),  is  defined  as 
follows 

1111 

W21©21  14^21021  14^22  0  22  W22022 
1111 
14/23  0  23  ’  44/23023  ’  W24024  ’  W24024 

Appendix  II 
Proof  of  Theorem 


Proof:  To  facilitate  the  proof,  a  nonnegative  Lyapunov 
function  V\  ( t )  £  R  is  defined  as  follows 

V4  4  X-eTZe  (59) 


where  Z  ( t )  was  defined  in  (15).  The  time  derivative  of  (59) 
is  given  as  follows 

Vx  =  eTZe  +  \eTZe.  (60) 

The  expression  in  (60)  can  be  written  as  follows 

Vi  =  -keTfmTe  +  eTW3Q.  (61) 


where  the  closed-loop  error  system  given  in  (28)  was  utilized. 

To  facilitate  the  analysis  another  nonnegative  Lyapunov 
function,  denoted  by  V2  ( t )  £  R,  is  defined  as  follows 

y2  4  (62) 

After  taking  the  time  derivative  of  (62)  and  substituting  (44) 
and  (48),  the  following  expression  is  obtained 

V2  =  — 0Tr_1Proj  {aVWTp  +  rW3re}  +  QTWTWQ. 

(63) 

After  utilizing  the  property  of  the  projection  (see  (116)  in 
Appendix  VI),  ifU)  can  be  upper  bounded  as  follows 

V2  <  -aOTWTp  +  QTWTWQ  -  0T4V3r e.  (64) 

The  expression  in  (64)  can  be  rewritten  as  follows 

V2  <  -apTF1~Tp  +  pTFfTFCp  -  0t4V3t e  (65) 

where  (43)  was  utilized.  The  right-hand-side  of  (65)  can  be 
upper  bounded 

V2  <  —aepTp  +  ppTp  —  QtW-[ e  (66) 

where  the  definition  of  F\  ( t )  in  (58)  is  utilized  along  with 
(34),  (46),  and  (47).  After  utilizing  the  definition  of  a  ( t )  in 
(45)  the  expression  in  (66)  can  be  written  as  follows 

V2  <  —e  ||p||2  —  QTW-[ e.  (67) 

To  prove  the  theorem,  the  following  nonnegative  Lyapunov 
function,  denoted  by  V  (t)  £  M  is  defined 

V  =  Vi +V2  (68) 


where  Vi  ( t )  and  V2  ( t )  were  defined  in  (59)  and  (62)  respec¬ 
tively.  The  time  derivative  of  (68)  can  be  upper  bounded  as 
follows 

V  <  —keT nnTe  —  £  ||p||2  (69) 

where  (61)  and  (67)  were  utilized.  From  (69),  the  following 
inequalities  can  be  written 

V  <  -fceTnnTe  (70) 

V  <  ~e\\p\\2.  (71) 


After  integrating  (70)  and  (71),  following  expressions  can  be 
obtained 


dr  <  V(to)  —  V'(+oo). 


(72) 


||p(r)||2  dr  <  V(t0)  -  ^(+00) 


(73) 


After  utilizing  (72),  (73)  and  the  fact  that  V(t)  is  nonneg¬ 
ative,  it  can  be  concluded  that  V(t)  <  V(fo)for  Vf.  Hence, 


V  (f)  is  bounded  (i.e.,  V  ( t )  G  Coo)-  Then,  from  (68)  it 
can  be  concluded  that  V\  (f),  V2  ( f )  G  £00  •  Since  V  (f)  is 
bounded  then  from  (72)  and  (73),  it  is  clear  that  nT  (f)  e  ( f ), 
p(t)  G  £2-  Since  Zi(t)  are  bounded  for  Vf  =  1,..,4  then 
from  (15)  it  follows  that  Z  (t)  G  Coo-  After  utilizing  the 
facts  that  V\  (t),  Z  ( t )  G  Coo  along  with  (59),  it  follows  that 
e  (f )  G  Coo ■  From  the  definition  of  e(t)  in  (16),  it  is  clear 
that  ei  (f)  €  Coo  Vi  =  1,  ..,4.  After  utilizing  the  fact  that 
the  desired  image  coordinates  pdi  Vi  =  1,  ..,4  are  constant 
along  with  the  fact  that  e*  (f)  Vi  =  1,  ..,4  are  bounded, 
then  from  (6),  it  is  clear  that  Pi{t)  G  Coo  Vi  = 

Based  on  Remark  3,  it  follows  that  q  (f)  is  bounded.  Since 
Pi  (t)  Vi  =  1,  ..,4  are  bounded  then  from  (9)  it  is  clear  that 
Aei(t)  G  Coo  Vi  =  1,  ..,4.  Since  Zi(t)  are  bounded,  from 
(33)  it  is  easy  to  see  that  W2  ( t )  G  Coo ■  After  utilizing  the 
facts  that  pi  ( t ),  W2  (t)  G  Coo  along  with  (35),  it  is  clear 
that  W\  {t)  G  Coo ■  By  using  (34)  along  with  (38),  it  can 
be  proven  that  IT)  [  0])  Oj,  ]T  G  £2-  From  (34),  it  is 
clear  that  F\  ( t )  is  bounded.  After  utilizing  the  above  facts 
along  with  (43),  it  can  be  concluded  that  W  (t)  0  (f)  G  £2- 
The  fact  that  V)  (f)  is  bounded  can  be  used  along  with 
(62)  to  show  that  0T(f) r_1(f)0(f)  G  Coo-  Since  F_1  (f) 
is  always  positive  definite  (see  Remark  6),  then  from  (62), 
it  is  clear  that  0(f)  G  Coo-  From  (26),  it  is  easy  to  see 
that  0(f)  G  Coo-  After  utilizing  the  above  boundedness 
statements  along  with  (37),  it  is  clear  that  pi  (f)  G  £00,  then 
from  (40),  it  is  easy  to  see  that  pi(t)  G  Coo-  The  above 
mentioned  boundedness  statements  can  be  used  to  prove 
that  W  (f)  is  bounded.  After  utilizing  the  above  mentioned 
boundedness  statements,  Remarks  3  and  4,  along  with  (52), 
(54),  and  (56);  then  from  (57),  it  is  clear  that  II  (f)  G  Coo- 
Since  II  (f)  is  a  function  of  0(f)  and  bounded  signals,  then 
it  is  bounded.  The  above  boundedness  statements  can  be 
utilized  along  with  (27)  to  show  that  u(t)  G  Coo-  Then, 
from  (21),  it  is  clear  that  W3  (f)  G  Coo-  Since  the  projection 
algorithm  provides  bounded  outputs  then  from  (44),  it  can  be 

concluded  that  0  (f)  G  Coo,  then  the  time  derivative  of  (26) 

can  be  used  to  show  that  0  (f)  G  Coo-  After  utilizing  the 
above  boundedness  statements  along  with  (13),  it  is  clear  that 
e,{  (f)  G  Coo  Vi  =  1,  ..,4.  From  (7),  it  is  clear  that  pi  (f)  G 
Coo-  Since  pt  (f)  Vi  =  1, ..,  4  are  bounded  then,  from  the  time 
derivative  of  (9),  it  is  clear  that  (f)  G  Coo  Vi  =  1, ..,  4. 
After  utilizing  the  above  boundedness  statements,  along  with 

the  fact  that  it  is  a  function  of  0  (f)  and  bounded  signals, 
it  is  easy  to  see  that  -Cf[(t)  G  Coo-  Now,  it  follows  that 

■C  ^IIT  (f)  e  (f)^  e  Coo-  Since  flT  (f)  e  (f )  G  C2  n  Coo,  it 

can  be  concluded  that  [27] 


nT  (t)e(f) 


0  as  f  — >  +00. 


(74) 


After  utilizing  Remarks  3  and  4  along  with  (14),  it  is  clear 
that  ij  (f)  G  Coo  Vi  =  1, 4;  hence,  from  the  time  derivative 
of  (33),  it  is  clear  that  W2  (f)  G  Coo-  After  utilizing  the  above 
boundedness  statements  along  with  the  time  derivatives  of 
(32)  and  (36),  it  can  be  proven  that  Wj  (f),  pi  (f)  € 
Coo,  hence  proving  that  W  (f)  G  Coo-  The  previously 


mentioned  boundedness  statements  can  be  utilized  to  prove 
that  -C  (lV(f)0(f))  e  Coo-  Since  IT(f)0(f)  £  £2  n£oo,  it 
can  be  concluded  that  [27] 


w(tm) 


0  as  f  — >  +00. 


(75) 


As  shown  in  Appendices  III  and  IV,  if  the  signals  11(f) 
and  W(t)  satisfy  the  persistent  excitation  (PE)  conditions 
given  in  (50),  (51),  then  from  (74)  and  (75)  it  can  further  be 
concluded  that 


Mt)| 

— >  0  as  f  — >  +00 

(76) 

0(f) 

— >  0  as  f  — >  +00. 

(77) 

Appendix  III 
PE  Proof  for  e  (f) 

An  auxiliary  function  1  (f 0  >  f )  £  R8x8  is  defined  as 


follows 


fir  =  /  n(r)nT(r)cfr. 


(78) 


To  facilitate  the  proof  the  following  expression  is  considered 

{eT  (r)  fti  (t0,  t)  e  (t)}  =  eT  (r)  Q1  (f0,  r)  e  (r) 

+eT  (t)  (fir  (fo,  r)}  e  (r)  +  eT  (r)  fli  (f0,  r)  e  (r) . 

(79) 

From  (79),  the  following  expression  can  be  obtained  after 
integrating  from  fo  to  f  and  rearranging  the  terms 

f  eT  (r)  fii  (f0,  r)  e  (r)  dr  =  eT  (f)  fli  (f0,  f)  e  (f) 

■1 10 

-eT  [t0)  Oi  (fo,  t0)  e  (f0)  -  f  eT  (r)  fii  (f0,  r)  e  (r)  dr 

Jtn 


-  [  eT  (r)  (fii  (f0,  r)}  e  (r)  dr. 

Jto  dT 

After  utilizing  the  following  facts 

^l(7o,^o)  =  08x8 

fif(f0,r)  =  Oi(f0,r) 

^{fit(fo,T)}  =  n  (t)  nT  (r) , 

the  expression  in  (80)  can  be  rearranged  as  follows 


(80) 


(81) 


eT  (f)  fli  (f0,  f)  e  (f) 

=  2  f  eT  (t)  fii  (f0,  t)  e  (r)  dr 
Jt0 

+  f  eT  (r)  n(r)nT(r)e  (r)  dr  (82) 

Jt.0 

where  0sX8  £  R8x8  is  a  matrix  of  zeros.  To  further  facilitate 
the  proof,  the  following  lemma  is  stated  [28]: 


Lemma  1:  Let  /  (t)  be  a  uniformly  continuous  function 
[27],  Then, 

lim  /  (f)  =  0  TT  lim  /  /  (r)  dr  =  0  (83) 

t — »+oo  t— »+oo  J ^ 

for  any  positive  constant  (  el. 

To  utilize  Lemma  1,  a  change  of  variables  is  applied  to 
(82)  by  substituting  t.  with  tg  +  T,  where  T  £  R  is  a  positive 
constant.  The  following  expression  is  obtained  after  applying 
a  limit  operation  to  the  resulting  equation 

lim  e ^  (t o  +  T)  fli  (to,  to  +  T )  e  (fo  +  T) 

to — >+oo 

rto+T 

=  2  lim  /  eT  (t)  0\  (to,  t)  e  (t)  dr 
io-^+oo  L 


lim 

io^+oo 


eT  (r)  LI  (r)  L1t  (r)  e  (r)  dr. 


Remark  7:  The  right-hand-side  of  (18)  can  be  written  as 
follows 

Ze  =  ~kB1JrflTe  (85) 

where  (27)  was  utilized.  After  utilizing  (74),  it  is  easy  to  see 
that  the  right-hand-side  of  (85)  goes  to  zero.  After  utilizing 
this  fact  along  with  the  fact  that  Z  ( t )  is  positive  definite, 
then  it  is  clear  that 

|| e  (t )  ||  — >  0  as  t  — >  -Too.  (86) 

After  utilizing  (50),  (86),  Lemma  1,  and  the  fact  that  e  ( t ) 
is  bounded,  then  it  is  clear  that  the  first  term  at  the  right-hand- 
side  of  (84)  is  equal  to  zero.  After  utilizing  (74),  Lemma  1, 
and  the  facts  that  11(f),  e  (f)  are  bounded,  then  it  is  clear 
that  the  second  term  at  the  right-hand-side  of  (84)  is  equal  to 
zero.  Thus  the  following  expression  can  be  obtained  based 
on  (84) 

lim  e ^  (to  +  T)  (to,  to  +  T)  e  (fo  +  T )  =  0.  (87) 

to— »+oo 

After  utilizing  the  fact  that  7j  1$  <  fii  (to,  to  +  T )  it  is  clear 
that 

|| e  (f)||  — *•  0  as  f  — >  +oo.  (88) 

Appendix  IV 
PE  Proof  for  0  (f) 

An  auxiliary  function  Q2(to,t)  G  Rpxp  is  defined  as 
follows 

02=  [  WT (t)W (r)dr .  (89) 

•I  to 

To  facilitate  the  proof  the  following  expression  is  considered 
-^-|eT  (r)02  (to,r)0(r)| 

=0  (r)ft2(to,r)0(r) 

+0T  (t)  3 -  {fl2  (t0,  r)}  0  (r) 


+0T  (r)  (t0,r)  0  (r) . 


From  (90),  the  following  expression  can  be  obtained  after 
integrating  from  to  to  f  and  rearranging  the  terms 

f  6T(T)f22(t0,r)  0  (r)dr 

Jto 

=  0T  (f)  fl2  (to,  t)  0  (f)  —  0T  (to)  fl2  (to,  to)  0  (to) 

-  f  0  (r)ft2(to,r)0(r)dr 
Jto 

-f  0T(r)-^-{ft2(to,r)}0(r)dr.  (91) 

Jto  dT 

After  utilizing  the  following  facts 

^2  (fo,  fo)  =  0 pxp  (92) 

(to,  r)  =  0.2  (to,  t) 

jL{n2(to,T)}  =  WT  (t)W(t), 

the  expression  in  (9 1 )  can  be  rearranged  as  follows 
0T  (t)  02  (to,t)0(t) 

=  2  [  0T  (r)  fl2  (t0,  r)  0  (r)  dr 


+  /  0T  (r)  W/T(r)W/ (r)0  (r)  dr  (93) 

Jto 

where  0pxp  €  Rpxp  is  a  matrix  of  zeros.  A  change  of 
variables  is  applied  to  (93)  by  substituting  t  with  t0  +  T. 
The  following  equation  is  obtained  after  applying  a  limit 
operation  to  the  resulting  equation 


lim  eT  (to+T)n2(to,to  +  T)Q(to  +  T) 

t0—>+oo 


=  2  lim 

to^+oo 


+  lim 

t0^+oo 


0T  (r)  02  (t0,  r)  0  (t )  dr 


0T  (r)  I+T  (r)  PL  (r)  0  (r)  dr.  (94) 


Remark  8:  The  right-hand-side  of  (44)  can  be  written  as 
follows 

0=  Proj  |  otVWT  —^——W Q  +  rW^ei  (95) 
(  n/202  J 

where  (38)  was  utilized.  From  (75),  it  is  clear  that  the  first 
term  inside  the  bracket  on  the  right-hand-side  of  (95)  goes 
to  zero,  and  from  (88)  the  second  term  inside  the  bracket 

goes  to  zero.  So,  both  0  and  0  (t)  go  to  zero  as  t  — >  +oo. 

After  utilizing  (51),  (95),  Lemma  1,  and  the  fact  that  0  (t) 
is  bounded,  it  is  clear  that  the  first  term  at  the  right-hand-side 
of  (94)  is  equal  to  zero.  After  utilizing  (75),  Lemma  1.  and 
the  facts  that  W  (r),  0  (r)  are  bounded,  it  is  clear  that  the 
second  term  at  the  right-hand-side  of  (94)  is  equal  to  zero. 
Thus  the  following  expression  can  be  obtained  based  on  (94) 

lim  eT  (to  +  T)n2(to,to+T)Q(to+T)=0.  (96) 

to— »  +  oo 

After  utilizing  the  fact  that  7 3Ip  <  f l2  (to,  to  +  T )  it  is  clear 
that 

II©  (i)||  — >  0  as  t  — >  +00. 


(90) 


(97) 


Appendix  V 

Extension  to  Fixed-Camera  Configuration 

In  this  section,  an  extension  to  fixed-camera  configuration 
is  presented.  The  analysis  for  the  fixed-camera  configuration 
varies  from  the  camera-in-hand  configuration  only  for  the 
open-loop  error  system  development.  The  rest  of  the  analysis 
are  same  for  both  configurations.  The  error  system  develop¬ 
ment  for  the  fixed-camera  case  is  presented  below. 

In  order  to  develop  a  geometric  relationship  between 
fixed-camera  and  moving  object,  an  orthogonal  coordinate 
frame,  denoted  by  T,  attached  to  the  object,  an  inertial 
coordinate  frame,  denoted  by  I,  whose  origin  coincides 
with  the  optical  center  of  the  fixed-camera,  and  an  inertial 
coordinate  frame,  denoted  by  W,  attached  to  the  base  frame 
of  the  robot  manipulator  are  defined  (see  Figure  2).  Let 
the  3D  coordinates  of  the  ith  feature  point  on  the  object 
be  denoted  by  the  constant  s,  6  R3  relative  to  the  object 
reference  frame  T ,  and  fhi  ( t )  G  R3  relative  to  the  inertial 
coordinate  frame  I,  which  is  defined  in  (1). 


(Image  plane) 


w 


(Base  of  the  robot  manipulator) 


Fig.  2.  Geometric  relationships  between  the  fixed-camera,  robot  manipu¬ 
lator  and  the  object  attached  to  its  end-effector. 

The  image  error  for  the  ith  feature  point,  denoted  by 
ei  ( t )  G  R2  is  defined  in  (6).  To  facilitate  the  error  system 
development  the  dynamics  of  the  image  error  is  found  as 
follows 

e-i  =  —AeiRc  [  Rr  RrS  (sj)  1  Jru.  (98) 

Zi 

In  (98),  Aei  ( t )  G  R2x3  is  a  function  of  camera  intrinsic  cal¬ 
ibration  parameters  and  image  coordinates  of  the  ith  feature 
point,  defined  in  (9),  Rc  G  SO  (3)  is  the  camera  extrinsic 
calibration  matrix,  Rr  (t)  G  SO  (3)  is  the  orientation  matrix 
of  the  end-effector  of  the  robot  manipulator,  Jr  ( q )  G  R6x6  is 
the  Jacobian  matrix  of  the  robot  manipulator,  and  u  ( t )  G  R6 
represents  the  kinematic  control  input.  It  should  be  noted 
that  Remarks  2,  3,  and  4  are  satisfied  for  the  fixed-camera 
configuration. 


Remark  9:  In  the  subsequent  analysis,  it  is  assumed  that 
Si  G  R3  are  known  (see  [29]  for  the  precedence  of  this  kind 
of  assumption). 

After  utilizing  (1)  and  (98),  the  dynamics  of  Zi  (t)  is 
obtained  as  follows 

Zi  =  rf3  [  Rr  RrS  ( Si )  ]  Jru  (99) 

where  rj3  G  Rlx3  is  the  third  row  vector  of  extrinsic  camera 
calibration  matrix. 

To  facilitate  the  subsequent  analysis  a  diagonal  matrix, 
denoted  by  Z  (t)  G  R8x8  with  its  entries  being  Z{  (t)  V*  = 
1,  ..,4,  and  a  combined  error  signal  e  (t)  G  R8  are  defined 
as  in  (15)  and  (16).  After  utilizing  (15),  (16),  and  (98),  the 
following  expression  can  be  obtained 


Ze  =  B\Jru  (100) 

where  auxiliary  signal  B\  ( t )  G  R8x6  is  defined  as  follows 


B  i  = 


AeiRc 

Rr 

RrS  (Si) 

Ae2i?c 

Rr 

RrS  (s2) 

Ae3Rc 

Rr 

RrS  (s3) 

Ae4Rc 

Rr 

RrS  (S4) 

(101) 


After  utilizing  (16)  and  the  time  derivative  of  (15),  Z  (t)  e  (t) 
can  be  written  as  follows 


Ze  =  Ei— [  zi 
at 


Z 1  Z2  Z2  Z3  Z3  Z\  24  J 

(102) 

x8  is  defined  as  follows 


where  diagonal  matrix  E\  (t)  G 

Ei  =  diag  {en,  ei2,  e2i,  e22,  e3i,  e32,  e4i,  642}  (103) 

with  (f)  being  the  jth  entry  of  e,  (f).  The  second  term  on 
the  right-hand-side  of  (102)  can  be  written  as  follows 


dt 


[  ZI  ZI  Z2  Z2  z3  Z3  Z4  Z4  ]T  =  CiJrU 

(104) 

where  auxiliary  signal  Ci  ( t )  G  R8x6  is  defined  as  follows 


Ci  = 


r  t 

rc3 

Rr 

RrS  (Si) 

T 

rc3 

Rr 

RrS  (Si) 

T 

C3 

Rr 

RrS  (S2) 

T 

C3 

Rr 

RrS  (s2) 

T 

rc3 

Rr 

RrS  (S3) 

T 

C3 

Rr 

RrS  (S3) 

T 

rc3 

Rr 

RrS  (S4) 

T 

L  rc3 

Rr 

RrS  (S4) 

(105) 


where  (99)  was  utilized. 

A  prediction  error  formulation  for  the  unknown  parameters 
will  be  introduced.  From  Figure  2,  fhi  (t)  can  be  written  as 
follows5 

fhi  =  Rcxr  +  RcRrSi  +pc  (106) 

where  pc  G  R3  is  the  position  of  the  origin  of  frame  X  with 
respect  to  frame  W  expressed  in  frame  W.  After  utilizing 


5For  the  derivation  of  the  expression  for  tn ;  the  reader  is  referred  to 
Appendix  VII. 


(1),  (2),  and  (4),  the  pixel  coordinates  for  the  ith  feature  point 
can  be  written  as  follows 

Pi  =  [RcXr  +  RcRrSi  +Pc]  (107) 

Z% 

where  the  corresponding  depth  can  be  written  as  follows 

Zi  =  r  J3xr  +  rJ.:iR,rSt  +  pc3  (108) 

where  r J  (t)  £  R3  is  the  last  row  of  R  (t).  It  should  be  noted 
that,  in  (107)  and  (108),  A,  Rc,  Si,  pc  are  unknown  constant 
parameters,  and  xr  (t),  Rr  (t)  are  measurable  signals  (see 
Remark  2).  The  rest  of  the  prediction  error  formulation  is 
same  as  the  camera-in-hand  configuration. 


Appendix  VI 
Projection  Algorithm 

The  positiveness  of  the  term  W2  ( t )  ©2  (t)  is  ensured  by 

a  projection  operator  on  0  (t)  [24].  To  facilitate  the  subse¬ 
quent  development,  an  auxiliary  scalar  function  is  defined  as 
follows 

V  (o\  =  e  -  TV,©,  (109) 


where  its  gradient  is  computed  as  follows 

Vgp(©)  =  [  0ixri  -W2]  (110) 

where  e  being  an  arbitrarily  small  positive  constant  and 
Oixr-i  £  Rlxri  being  a  vector  of  zeros.  Two  convex  sets 
based  on  the  function  V  ^0^  are  defined  as  follows 


1Z  =  |0G 
Us  =  (ee 


:  V  (©)  <  0} 
:  V  (©)  <  5} 


where  6  £  R  is  a  positive  constant  that  is  very  close  to  zero. 
Let  the  boundary  and  the  interior  of  set  TZ  be  defined 

O 

by  dlZ  and  TZ  ,  respectively.  Based  on  these  definitions,  the 
projection  of  r  (t)  is  defined  as  follows 


Proj{r}4 


where  PT  (t)  £ 


T  0  £lZ  or  VqPtt  <  0 
PT  0  £  TZS\  R  and  V qVtt  >  0 


is  defined  as  follows 


C) 

where  the  auxiliary  scalar  function  c  ^0^  i 
follows 

c(e)Amill|1,^l|. 


is  defined  as 


=  min  <  1, 


It  is  helpful  to  note  that  c(dTZ  )  =  Oand  c(dTZs)  =  1-  The 
suggested  projection  operator  satisfies  the  following  property 
(reader  is  referred  to  [24]  for  the  proof) 


Appendix  VII 
Derivation  of  ifii  ( t ) 

A.  Camera-In-Hand  Configuration 

In  Figure  1,  let  the  vector  representing  the  distance  be¬ 
tween  the  origin  of  £  and  Oi  be  denoted  as  Xi  ( t )  £  R3. 
From  the  triangle  formed  by  O,  and  the  origins  of  T  and  £, 
it  is  easy  to  see  that 

fhi  =  RcXi  +  pc-  (117) 

Similarly,  from  the  triangle  formed  by  Oi  and  the  origins  of 
W  and  £,  the  following  can  be  obtained 

xPi  =  RrXi  +  xr.  (118) 

After  solving  (118)  for  X{  (t)  and  substituting  the  resulting 
expression  into  (117),  we  obtain 

rrii  —  Rc  [Rr  ( Xpi  Xjf  T  Pc\  •  (119) 

B.  Fixed-Camera  Configuration 

In  Figure  2,  from  the  triangle  formed  by  Oi  and  the  origins 
of  T  and  I,  it  is  easy  to  see  that 

fhi  =  RcRrSi  +  Xf.  (120) 

Similarly,  from  the  triangle  formed  by  the  origins  of  W,  X 
and  T,  the  following  can  be  obtained 

Xf  =  Rcxr  +  pc.  (121) 

After  substituting  x i  (t)  found  in  (121)  into  (120),  we  obtain 

fhi  =  Rcxr  +  RcRrSi  +  pc-  (122) 


— 0Tr_1Proj  {r}  <  v©  g  tz5,  ©  g  tz. 

(116) 


